Multi-image capture system with improved depth image resolution

ABSTRACT

The multi-image capture system comprises an imaging lens, a beamsplitter directing a first portion of light on an array of micro-imaging elements and a first photodetector array divided into a plurality of macropixels providing a set of depth images, and another portion of light on a second photodetector array  6  providing a main image with a resolution being higher than the resolution of each basic depth image. According to the invention, the resolution of the depth images is increased by using the image data of the main image.

TECHNICAL FIELD

The invention relates to plenoptic image capture devices that are ableto provide a depth map of objects in an object field.

BACKGROUND ART

The document U.S. Pat. No. 5,076,687 discloses a multi-image capturesystem comprising:

a converging lens able to receive light from objects in an object fieldand to direct said received light to an image field of the converginglens;

a plurality of micro-lenses distributed about the image field of theconverging lens such that each micro-lenses receives light from theconverging lens and forms an image from the light it receives;

a photodetector array divided into a plurality of macropixels each ofwhich is subdivided into a plurality of pixels, each macropixelreceiving an image formed by one of the micro-lenses such that eachpixel of a macropixel receives light from one portion of the converginglens as imaged by one of the micro-lenses, each pixel being able togenerate an image data indicative of the intensity of light incidentupon it;

and a data processor able to build a plurality of so-called“depth-images” of said objects in the object field, such that each imageis built from image data generated by pixels having like spatialpositions within the different macropixels of the photodetector array.

See also document DE 3018470 (SIEMENS) where the capture systemcomprises also a beamsplitter positioned on the optical path between theimaging lens and the array of micro-imaging elements to redirect asecond portion of the light received from the imaging lens to a secondimage field (ref. 33) of the imaging lens.

See also document US 2006/221209 disclosing a tree diagram ofacquisition of a plenoptic field originating from a scene, where nodeswith multiple child nodes of this diagram represents beam splitters andleaf nodes represent image sensors.

The data processor is also able to compare the parallax between thedifferent depth-images in order to determine depth of said objects inthe object field.

Each depth image is representative of light passing through a differentregion of the converging lens, and then gives an image of the objectfield under a different angle of view.

Generally:

there is one macropixel for each image generated by a micro-lens;

each macropixel comprises the same number of pixels;

in all macropixels, the pixels are spatially distributed similarly;

the microlenses are arrayed in a rectangular lattice, corresponding tothe format of each of the plurality of depth images.

In a different image capture device, U.S. Pat. No. 7,199,348 proposesalso to use the parallax that has been determined between differentviews of a same object field to map the depth of the objects within thisfield.

As disclosed in U.S. Pat. No. 5,076,687, the number of pixels in eachmacropixel determines the number of different depth images though theconverging lens of the multi-image capture system, and the number ofmicrolenses determines the spatial resolution of these depth images.

Presently, the number of pixels in the photodetector array is alimitation for the spatial resolution of the depth map of the objectfield that can be obtained by comparing the parallax between thedifferent depth-images. In the article entitled “Light Field Photographywith a Hand-held Plenoptic Camera”, authored by Ren Ng, Marc Levoy,Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan, published inthe Standford Technical Report CTSR, on February 2005, a KodakKAF-16802CE photodetector array of 4000×4000 pixels is used inconjunction with an array of 296×296 microlenses. But using such highresolution photodetector array is very expensive although the resolutionof each of the depth image (i.e. 296×296) remains unacceptably low.

SUMMARY OF INVENTION

It is an object of the invention to improve the spatial resolution ofthe depth map without changing the number of pixels in the photodetectorarray.

For this purpose, the subject of the invention is a multi-image capturesystem comprising:

an imaging lens able to receive light from objects in an object fieldand to direct a first portion of said received light to a first imagefield of the imaging lens;

an array of micro-imaging elements distributed about the image field ofthe imaging lens such that each micro-imaging element receives lightfrom the imaging lens and forms a partial image of said objects from thelight it receives;

a first photodetector array divided into a plurality of macropixels eachof which is subdivided into a plurality of pixels, each macropixelreceiving an image formed by one of the micro-imaging elements such thateach pixel of a macropixel receives light from one portion of theimaging lens as imaged by one of the micro-imaging elements, each pixelbeing able to generate an image data indicative at least of theintensity of light incident upon it;

and a data processor able to build a plurality of so-called basic“depth-images” of said objects in the object field, such that each basicdepth-image is built from image data generated by pixels having likespatial positions within the different macropixels of the firstphotodetector array,

a beamsplitter positioned on the optical path between the imaging lensand the array of micro-imaging elements to redirect at least a secondportion of the light received from the imaging lens to at least a secondimage field of the imaging lens,

at least one second photodetector array divided into a plurality ofpixels receiving an image of said objects formed in the second imagefield, each pixel being able to generate an image data indicative atleast of the intensity of light incident upon it;

wherein said data processor is also able to build a so-called “mainimage” of said objects in the object field from image data generated bypixels of the at least second photodetector array.

As each basic depth-image is built from image data generated by pixelshaving like spatial positions within the different macropixels of thefirst photodetector array, it means that the number of pixels in eachbasic depth-image is equal to the number of macro-pixels within thefirst photodetector array.

Preferably, the number of pixels of the at least second photodetectorarray that are used to build said “main image” is superior to the numberof macropixels of the first photodetector array that are used to buildeach basic depth image. It means that the basic resolution of the mainimage is superior to the basic resolution of each basic depth image.

Preferably, by using said main image, said data processor is also ableto upconvert said basic depth images to upconverted depth images havinga number of pixels superior to the number of macropixels of the firstphotodetector array. Consequently, it is possible to upscale the lowresolution of the basic depth images by using the highest resolution ofthe main image.

Preferably, said data processor is also able assign a basic disparitymap to each of said basic depth images. There are several known methodsfor determining a disparity map associated to a basic depth image,compared for instance to a central basic depth image. For instance,interrogation regions corresponding to different objects of a centralbasic depth image are selected, and candidate regions in the depth imageare compared to the interrogation region. The candidate region which ismost similar to the interrogation region is identified as a matchingregion. The location of the matching region within the depth image,relative to the location of the interrogation region within the centralbasic depth image, specifies the disparity for the interrogation region.Conventional correlation techniques are generally used for determiningthe similarity of a candidate region and an interrogation region.

Preferably, said data processor is also able to interpolate from anybasic disparity map an interpolated disparity map having a higherresolution.

Preferably, said data processor is also able to calculate eachupconverted depth image of a basic depth image from the main image andthe interpolated disparity map associated with this basic depth image.

Preferably, said data processor is also able to compare the parallaxbetween the different up-converted depth-images in order to determinedepth of said objects in the object field. To determine this depth, asingle baseline iterative registration algorithm can be used forinstance to calculate the depth for each pixel element, as disclosed inthe article entitled “An Iterative Image Registration Technique with anApplication to Stereo Vision”, by B. D. Lucas & Takeo Kanade, publishedin the Proceedings of Imaging Understanding Workshop, pp. 121-130(1981), or a more precise algorithm using multi-baselines, as disclosedfor example in the article entitled “Multi-Baseline Stereo using aSingle-lens Camera”, by M. Amtoun & B. Boufama, published in 2003 in theproceedings of the International Conference on Image Processing.

Preferably, in order to be able to compare the parallax between thedifferent up-converted depth-images, said data processor is also able toperform, across said first image field, a plurality of localdisplacement estimates of image displacement between up-converted depthimages.

As the resolution of the depth images has been increased by using theimage data of the main image, the resolution of the depth map that isobtained is also advantageously increased.

With this at least dual channel principle where a portion of the visiblelight is directed toward a plenoptic photodetector array, and whereanother portion of the visible light is directed toward a classical highdefinition photodetector array, we have a way to shoot one full HighDefinition image, and with the help of that High Definition image, tocalculate the depth map at full resolution for each image byinterpolating the different views from the plenoptic channel. Hence,this is a way of delivering an image and its associated depth map with aplenoptic video system without sacrificing the resolution.

Preferably, the plurality of micro-imaging elements comprises an arrayof microlenses. The microlenses or the micro-imaging elements may behomogeneously distributed over the first photodetector array, ordistributed as disclosed for instance in the document U.S. Pat. No.6,137,535.

In summary, the multi-image capture system according to the inventioncomprises an imaging lens, a beamsplitter directing a first portion oflight on an array of micro-imaging elements and a first photodetectorarray divided into a plurality of macropixels providing a set of depthimages, and another portion of light on a second photodetector arrayproviding a main image with a resolution being higher than theresolution of each basic depth image. According to the invention, theresolution of the depth images is increased by using the image data ofthe main image.

BRIEF DESCRIPTION OF DRAWINGS

The invention will be more clearly understood on reading the descriptionwhich follows, given by way of non-limiting example and with referenceto the appended figures in which:

FIG. 1 is a schematic diagram of a multi-image capture system accordingto a main embodiment of the invention;

FIG. 2 illustrates the general principle of the plenoptic imaging whichis used in the multi-image capture system of FIG. 1;

FIG. 3 illustrates the demultiplexing process that is implemented by themulti-image capture system of FIG. 1 to build a plurality of basic“depth-images” of the same objects 1 in an object field.

FIG. 4 illustrates how each basic depth image is a slightly non uniformshifted version of the central basic depth image or of the main image,all said images being provided by the multi-image capture system of FIG.1.

The figures take no account of the scale of values so as to betterreveal certain details, which would not be clearly apparent if theproportions had been respected.

DESCRIPTION OF EMBODIMENTS

A main embodiment of the multi-image capture system according to theinvention will now be described in reference to FIGS. 1 and 2. Themulti-image capture system comprises:

an imaging lens 2 able to receive light from objects 1 in an objectfield and to direct a first portion of the received light to a firstimage field of the imaging lens through a first output of a beamsplitter3, and to direct a second portion of the received light to a secondimage field of the imaging lens 2 through a second output of thebeamsplitter 3;

an array 4 of micro-imaging elements 41 distributed about the imagefield of the imaging lens such that each micro-imaging element receiveslight from the imaging lens and forms a partial image of the objects 1from the light it receives; each micro-imaging element 41 is generally amicrolens; a combination of microlenses and prisms can also be used asdescribed in US2007-230944.

a first photodetector array 5 divided into a plurality of macropixels 51each of which is subdivided into a plurality of pixels 511, eachmacropixel receiving an image formed by one of the micro-imagingelements 41 such that each pixel of a macropixel receives light from oneportion of the imaging lens as imaged by one of the micro-imagingelements;

a second photodetector array 6 divided into a plurality of pixelsreceiving an image of the objects 1 formed in the second image field;

the beamsplitter 3, already mentioned, which is then positioned on anoptical path between the imaging lens 2 and the array 4 of micro-imagingelements, and also positioned on an optical path between the imaginglens 2 and the second photodetector array 6.

FIG. 2 illustrates how a point on an object 1, which is seen by theimaging lens 2 under different parallaxes, can be recorded on the firstphotodetector array 5 without averaging on all the directions of theincoming light rays. Each pixel 511 under the same micro-imaging element41, i.e. belonging to the same macropixel, records the light rays comingfrom a different direction. The different spatial positions of thepixels within the same macropixel correspond to different directions ofthe incoming light rays.

In this specific embodiment, the first and the second photodetectorarrays are CCD devices of 4000×2040 pixels. CMOS devices can be usedinstead. Each pixel is able to generate an image data indicative of theluminance and of the chrominance of light incident upon it. Each imagedata is then a usual RGB data in the device-dependant color space.

In this specific embodiment, the array 4 comprises 800×680 micro-imagingelements; consequently, each macropixel of the second photodetectorarray comprises 5×3 pixels.

The multi-image capture system also comprises a data processor which isable:

to build a so-called “main image” of the objects 1 in the object fieldfrom image data generated by pixels of the second photodetector array 6;such a main image is provided in a manner known per se as in the mostcommon color image capture device using CCD; the resolution of this mainimage is 4000×2040;

to build a plurality of so-called basic “depth-images” of the sameobjects 1 in the object field from image data generated by pixels of thefirst photodetector array 5, such that each basic depth-image is builtfrom image data generated by pixels having like spatial positions withinthe different macropixels of the first photodetector array 5; as eachmacropixel comprises 5×3=15 pixels having different spatial positionswithin this macropixel, as the array 4 comprise 800×680 micro-imagingelements, 5×3=15 basic depth-images are then provided, each of themhaving a resolution of 800×680. As each individual pixel under onemicro-imaging element records one specific parallax, a parallaxdemultiplexing is performed as disclosed in U.S. Pat. No. 5,076,687 byseparating image data from the different pixels and concatenating into asingle basic depth image all image data from pixels having like spatialpositions within the different macropixels; such a demultiplexingprocess is illustrated on FIG. 3.

The angle of view of the main image is zero, corresponding to theaverage angle of view of the 15 basic depth images; the angle of view ofthe main image corresponds to the angle of view of the central basicdepth image.

In this specific embodiment, the number of pixels 4000×2040 of thesecond photodetector array that are used to build the “main image” issuperior to the number of macropixels 800×680 of the first photodetectorarray; consequently, the resolution 800×680 of each of the basic depthimages is far inferior to the resolution 4000×2040 of the main image.

We will now describe how the resolution of each of the depth images canbe improved by using image data from the highest resolution main image.

Each basic depth image is a slightly non uniform shifted version of thecentral basic depth image or of the main image. By non uniform shifting,it is understood that point objects are shifted differently according totheir relative position with respect to the focusing plane in the objectfield of the imaging lens 2. Points that are in the plane of focusingwill map at the same spatial position within the different macropixelsof the first photodetector array 5 in each basic depth image. If we nowconsider one particular basic depth image, other than the central one,objects of the object field that are in front of this focusing planewill undergo an opposite direction shift with respect to objects thatare behind this focusing plane. Such a shift is illustrated on FIG. 4.On the left top part of this figure, an image of an object point that islocated in front of the focusing plane is illustrated. On the right toppart of this figure, an image of another object point that is locatedbehind the plane of focusing is illustrated. In the bottom parts of thisfigure, the corresponding parallax de-multiplexed basic depth images areillustrated. One can see that, at the left bottom image number V(2,5),the image of that pixel is shifting toward left as it is indicated bythe arrow. On the contrary, the same depth image V(2,5) on the bottomright part is shifting in the opposite direction.

Step 1: Disparity Estimation

According to the invention, the data processor is also adapted tocalculate a basic disparity map associated with each basic depth imagein reference to one of these basic depth images.

Let us consider two images with image data being ordered, as usual, byhorizontal and vertical location. If a same object as an apple appearsas a sub-image in both images, but at different locations in each, thedisparity associated with the sub-image of the apple is the differencebetween these locations. Given similar sub-images of a same object in anobject field, a basic disparity map can be constructed which specifieswhere each region of a first sub-image appears in a second sub-image,relative to the location of the region in the first sub-image.

Each basic depth image of the set of 15 basic depth images that isprovided by the multi-image capture system according to the inventiontypically contains representations of many of the same objects as theother basic depth images. Although the objects are viewed from slightlydifferent perspectives, the representation of each object is generallysimilar in all these basic depth images. Due to the effect of parallax,the position of each object is usually different in each basic depthimage. By measuring the disparity associated with objects in the set of15 basic depth images, the depth of those objects can be determined. Bymeasuring the disparity associated with all small regions in the set of15 basic depth images, a basic disparity map can be determined for eachbasic depth image.

Let us for instance take the central basic depth image V(2,3) as thereference basic depth image to calculate the disparities. Let us furthertake any single V′ of the remaining 14 different depth images. Eachpixel of V(2,3) can be found in V′ by a block matching algorithm. It isfound at the same coordinates if the object at that point is in focus,and shifted if it is not. The amount of shift is named “disparity”. Sothat each depth image has all its 800×680 pixels assigned with adisparity.

There are several known methods for determining disparity from differentsets of data, as, for instance, from different images. Generally,interrogation regions (=windows) of a predetermined size (see EP686942,U.S. Pat. No. 5,867,591) or of a variable/adaptative size (see U.S. Pat.No. 6,141,440) are selected from a reference data set, as for instancefrom the central basic depth image, and candidate regions in a targetdata set, as for instance any other basic depth image, are compared tothe interrogation region. The candidate region which is most similar tothe interrogation region is identified as a matching region. Thelocation of the matching region within the target data set, relative tothe location of the interrogation region within the reference data set,specifies the disparity for the interrogation region. Conventionalcorrelation techniques are generally used for determining the similarityof a candidate region and an interrogation region.

With known block matching algorithms (BMA), it is then possible tocalculate the disparity for each pixel of the basic depth images, with asub-pixel resolution of a quarter of a pixel size. According to theinvention, a BMA is used to locate matching pixels in the set of 15basic depth images for purpose of disparity estimation. Each basic depthimage is then assigned with a basic disparity map. As the disparity isevaluated with a precision of a quarter of a pixel, it is sound tointerpolate each basic depth image up to four times its originalresolution of 800×680. The associated basic disparity map would stillhave a precision of one pixel.

Step 2: Upconversion of Each Depth Image to a Higher Resolution.

A disparity map of each basic depth image is then used to recalculateeach depth image from the high resolution main image, then providing anupconverted depth image with a higher resolution. For such arecalculation, each image data corresponding to a pixel of the mainimage is shifted from the disparity associated with the same pixel inthe basic depth image to upconvert.

But, as the number of pixels 4000×2040 of the second photodetector arraythat are used to build the “main image” is superior to the number ofmacropixels 800×680 of the first photodetector array that are used tobuild all basic depth images, only 800×680 pixels of the 4000×2040pixels of the main image can be associated with a disparity value ineach basic depth image.

To be able to associate each pixel of the main image with a disparityvalue for each basic depth image, an interpolation step should be addedthat is adapted to interpolate any disparity value associated to a pixelof the main image which has no correspondent in this basic depth imagefrom disparity values associated to neighbored pixels of the main imagewhich have correspondent in this basic depth image.

After this interpolation step, an interpolated disparity map is assignedto each basic depth image, having a resolution which is now identical tothe resolution of the main image.

Each basic depth image is then recalculated into an upconverted depthimage as follows: to each pixel of the main image is assigned adisparity value from the interpolated disparity map which is assigned tothis basic depth image, and a new image data is calculated by shiftingthis pixel from the said disparity value.

The same process is repeated for each basic depth image, then providinga set of 15 upconverted depth images.

As illustrated above, the data processor is then also able to upconvertthe basic depth images to upconverted depth images having a number ofpixels superior to the number of macropixels of the first photodetectorarray, wherein this upconversion uses image data of the main image.

As a variant to the main embodiment of the invention as described above,the multi-image capture system may comprise, instead of only one secondphotodetector array 6 to build the main image, three photodetectorarrays to build primary main images, one for each primary color as red,green and blue. The three primary main images are combined in a mannerknown per se to build the main image itself.

It can thus be appreciated that the present invention improves upon theprior art by providing a multi-image capture system allowing theimprovement of the spatial resolution of the depth map without changingthe number of pixels in the first photodetector array.

It will be understood that the present invention has been describedpurely by way of example, and modifications of detail can be madewithout departing from the scope of the invention.

Features disclosed in the description may, where appropriate, beimplemented in hardware, software, or a combination of the two.Reference numerals appearing in the claims are by way of illustrationonly and shall have no limiting effect on the scope of the claims.

While the present invention is described with respect to particularexamples and preferred embodiments, it is understood that the presentinvention is not limited to these examples and embodiments. The presentinvention as claimed therefore includes variations from the particularexamples and preferred embodiments described herein, as will be apparentto one of skill in the art. While some of the specific embodiments maybe described and claimed separately, it is understood that the variousfeatures of embodiments described and claimed herein may be used incombination.

1. A multi-image capture system comprising: an imaging lens able toreceive light from objects in an object field and to direct a firstportion of said received light to a first image field of the imaginglens; an array of micro-imaging elements distributed about the imagefield of the imaging lens such that each micro-imaging element receiveslight from the imaging lens and forms a partial image of said objectsfrom the light it receives; a first photodetector array divided into aplurality of macropixels each of which is subdivided into a plurality ofpixels, each macropixel receiving an image formed by one of themicro-imaging elements such that each pixel of a macropixel receiveslight from one portion of the imaging lens as imaged by one of themicro-imaging elements, each pixel being able to generate an image dataindicative at least of the intensity of light incident upon it; and adata processor able to build a plurality of so-called basic“depth-images” of said objects in the object field, such that each basicdepth-image is built from image data generated by pixels having likespatial positions within the different macropixels of the firstphotodetector array, a beamsplitter positioned on the optical pathbetween the imaging lens and the array of micro-imaging elements toredirect at least a second portion of the light received from theimaging lens to at least a second image field of the imaging lens, atleast one second photodetector array divided into a plurality of pixelsreceiving an image of said objects formed in the second image field,each pixel being able to generate an image data indicative at least ofthe intensity of light incident upon it; wherein said data processor isalso able to build a so-called “main image” of said objects in theobject field from image data generated by pixels of the at least secondphotodetector array, the number of pixels of the at least secondphotodetector array that are used to build said “main image” is superiorto the number of macropixels of the first photodetector array that areused to build each basic depth image, by using said main image, saiddata processor is also able to upconvert said basic depth images toupconverted depth images having a number of pixels superior to thenumber of macropixels of the first photodetector array.
 2. A multi-imagecapture system according to claim 1 wherein, for the upconversion ofsaid basic depth images, said data processor is also able: to assign abasic disparity map to each of said basic depth images, to interpolatefrom any basic disparity map an interpolated disparity map having ahigher resolution, and to calculate each upconverted depth image of abasic depth image from the main image and from the interpolateddisparity map associated with this basic depth image.
 3. A multi-imagecapture system according to claim 2 wherein said data processor is alsoable to compare the parallax between the different up-converteddepth-images in order to determine depth of said objects in the objectfield.
 4. A multi-image capture system according claim 1, wherein theplurality of micro-imaging elements comprises an array of microlenses.